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ABSTRACT 

Three studies identifying the characteristics of 
program submittals approved by the Joint Dissemintion Review Panel 
were examined. Between 1978-1985, 358 demonstration programs were 
submitted. For each of these years, between 51 percent and 83 percent 
of the submittals were approved. In this study, the submittals were 
reviewed on the basis of the Joint Panel* s criteria for program 
effectiveness; evidence of impact; evidence of statistical 
reliability of effects; evidence that the effects are educationally 
meaningful; interpretability of tests; credibility of evidence; 
evidence that the effects are attributable to the intervention; and 
evidence of generalisability. The reviewers also considered the 
clarity and quality of the submittal itself. The program content 
areas most frequently included were basic skills (reading, 
mathematics, and writing), career education, and special education. 
Approved submittals, in addition to the official seven criteria, also 
provided adequate descriptions of the program components. Most 
approvals used quasi -experimental or experimental designs, and 
nonequivalent or randomised control groups. Although the number of 
submittals decreased, evaluation designs became more rigid in more 
recent years. Clear writing and presentation of results appeared to 
be a critical factor in acceptance. (GDC) 
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The Joint Dissemination Review Panel is a federal board, consisting of 
members of the Department of Education and the National Institute of 
Education. Since its establishment in 1973, the Panel has been responsible 
for reviewing educational products and practices on the basis of effectiveness 
criteria. 

This paper presents information en three studies which were conducted on 
the 358 submittals which were reviewed by the Panel during the period 1978- 
85. The purpose of these studies was to identify those characteristics which 
distinguished approved submittals from disapproved ones. Each researcher 
examined the submittals on the basis of the Panel's effectiveness criteria. 
These effectiveness criteria were operationalized into variables so that the 
researcher could determine the presence or absence of the variable for each 
submittal. 

Results indicated that one of the key variables in determining whether a 
submittal is approved is the quality of the presentation of the program 
components and the evaluation findings. Other predictors of approval included 
the type of evaluation design which was implemented and the cost of 
installation. 
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The purpose of this paper Is to present information related to variables 
that nay be used in differentiating between submittals which are approved or 
rejected by the Joint Dissemination Review Panel ( JDRP) . Three primary 
sources of information were used: a dissertation completed by Wei Li Bang in 
1981, a dissertation completed by Kathleen Lynch in 1986, and a study 
completed by George Lam in 1986. Fang's dissertation covered submittals 
reviewed by JDRP in 1978 and 1979; Lynch* s dissertation covered 1980-1983; and 
Lam's study covered 1984 and 1985. A total of 358 submittals were reviewed by 
the Panel during this eight-year time period. The number of submittals for 
each of these eight years is presented in Table 1 below. 
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Table l 

Mabsr of Approvals and Disapprovals by Year 



1976-1985 

Resubmittal Rssubnsittal 
Approvals Disapprova ls Approved Rejected Total 





n 


* 


n 


% 


n 


* 


n 


* 


n 


1978 


23 


55 


14a 


33 


4 


10 


1 


2 


42 


1979 


29 


55 


15a 


28 


9 


17 


0 




53 


1980 


31 


69 


14 


31 


0 




0 




45 


1981 


42 


62 


24 


35 


1 


1 


1 


1 


68 


1982 


31 


51 


25 


41 


4 


7 


1 


2 


61 


1983 


30 


52 


20 


34 


5 


9 


3 


5 


58 


1984 


13 


68 


6 


32 


0 




0 




19 


1985 


10 


83 


_2 


12 


_0 




0 




1? 


Totals 


209 


58 


120 


34 


23 


6 


6 


2 


358 



Eleven of the 29 submittals which were reviewed in 1978 and 1979 were 
subsequently resubmitted and approved. The remaining 18 disapprovals were not 
resubmitted. 
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Table 1 Indicates that the number of submittals to the Panel peaked in 
1981, with a total of 68 submittals reviewed. In the last two veers there has 
been a notable decrease in the number of submittals. According to the JDRP 
Executive Secretary, this has been due to d e cr e as e d funding of the National 
Diffusion Network. Corresponding to the decrease in the number of submittals 
is an increase in the percentage of submittals approved. The Executive 
Secretary noted that the preparation and pres e n tation of submittals have 
improved in the past two years. 

In these three studta one of the major research questions was to 
identify those characteristics which distinguished approved submittals from 
those which ware disapproved. In order to collect relevant data, JDRP 
effectiveness criteria were reviewed and operational ized into specific 
variables. A data collection form was then developed which contained these 
variables. 

Effectiveness Criteria 

According to the Ideabook , there are seven effectiveness criteria by 
which submittals are Judged. These state that the submitting project has 
provided evidence of positive Impact, that effects are statistically 
significant and educationally Important, that measures are reliable and valid, 
that the observed effects can be attributed to the intervener that the 
evidence is credible, and that the intervention and its effec- -an be 
replicated in other sites. These criteria were operational ized in the 
following ways: 
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JDRP Criterion 

1. Evidence of impact 

2. Evidence of statistical 
reliability of effects 

3. Evidence that the effects are 
educationally meaningful 

4. Interpretability of 
measures 

5. Credibility of evidence 



6. Evidence that the effects 
are attributable to the 
intervention 

7. Evidence of generalizability 



Operational Definition of Variable 

* Evaluation design 

* Type of comparison group 

* Statistical significance 

* Educational importance 

* Type of instrument (standardized 
or locally developed) 

* Reliability of the instrument 

* validity of the instrument 

* Sampling procedures 

* Test administration procedures 

* Types of reported scores 

* Types of data analysis procedures 

* Internal validity 

* External validity 

* Replication 



In addition to these seven criteria, another variable was added. This 
addressed the clarity or quality of the submittal itself. 

Methodology in the three studies was similar. Each submittal was 
reviewed, and information on the seven JDRP criteria and two clarity variables 
was recorded on the data collection forms. Frequency data were tabulated on 
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each of the instruaent's items. In the pages that follow, selected variables 
will be presented for the three studies. 
Characteristics of Programs 

Table 2 presents the number of progr am s by content area. The majority of 
submittals was in the area of basic skills (e.g. , reading, math, writing) . Of 
the 428 submittals reviewed by Fang and Lynch, over half (i.e. , 224) addressed 
basic skills. 1 Two other areas that received attention were career education 
(48 submittals) and special education (36 submittals) . 



1 Even though there was only a total of 232 submittals in *0-83, Lynch 
coded some submittals as having more than one content area. she 
reported 345 programs fox the 232 submittals. 
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labia 2 

of Program a by 
Content Area 
Fang 

Apprw- Rejec- Resub- 
als tigns mlttals 



Lynch 
Approve Rejec- 
als tions 



Basic skills 
Career education 
Computer literacy 9 
Natural science* 
Social Science 9 
Special Education 
Health/Phys. Bd. a 
Gifted Education 3 
Arts/Humanities 3 
Early Childhood 

Education 
Teacher Education 
Other 



15 (29) b 1 (6) 3 (23) 
9 (17) 4 (22) 2 (15) 



7 (13) 



1 (8) 



8 (15) 5 (28) 2 (15) 
4 (8) 4 (22) 2 (15) 

9 (17) 4 (22) 3 (231 



139 (63) 66 (53) 
14 (6) 19 (15) 



11 (5) 
11 (5) 
20 (9) 
11 (5) 
4 (2) 
3 (1) 



8 (6) 

6 (5) 

8 (6) 

4 (3) 

3 (2) 
0 



0 4 (3) 

8 (4) 6 (51 



124° 



TOTAL 52 18 13 221° 

Note: This information was not recorded by Lam. 
a Thls was not a category used by Fang. 

b Numbers in parentheses reflect percentages of column totals. 
c Some submittals ware coded In more than one content area. There was a 
total of 346 programs for 232 submittals. 
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The majority of programs that were submitted to JDRP during the eight- 
year period were for students from kindergarten through twelfth grade. These 
programs represented 87* of the submittals reviewed. This was a trend 
noticeable in each of the three studies. 



Table 3 
Number of Progp— by 
Educational Level 





Preschool 


K-12 


Post-Secondarv 


Total 


Fang 


12 (14) a 


64 (77) 


7 (8) 


83 


Lynch 


12 (4) 


299 (90) 


23 (7) 


334 b 


Lam 




18 (95) 


_1 (5) 


19 


TOTAL 


24 


381 


31 


436 



lumbers in parentheses reflect percentages of row totals. 
b Some submittals r eporled more than one educational level. 

Even though JDRP emphasizes the seven effectiveness criteria, another 
essential ingredient of the submittal is a description of the components that 
comprise a program. When Fang reviewed submittals, she recorded whether none, 
some, or all of the components of the program were adequately described. 
Lynch used a somewhat different rating scale— she looked at whether some, 
most, or all of the components were described. These results are presented in 
Table 4. The majority of approved submittals either had most or all program 
components described. 
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Table 4 

ltaeber of Progre— Milch Incluk 
Dee crlptians of n ^* "tii'tt 



Coopanents Described 
!£n§ Some Most 6 All Total 



Approvals l a (2) c 23 (44) 

Rejections 15 (83) 
Rejections Later 

Resubmitted 12 (92) 
Resubmittal 

Approved 5 ( 46 ) 
Lynch 

Approvals 34 (24) 55 (38) 



28 (54) 52 

3 (17) 18 

1 (8) 13 

7 (54) 13 



55 (38) 144 

Rejections _34 (39) _39 (44) _15 (17) _88 

!24 94 104 328 



TOTALS 



Note: Lam did not look at this variable. 

"Wiile this project provided general information on the evaluation design, 
specific information on the intervention's features was presented. 
b This category was not used in Ring's study. 
c Numbers in parentheses reflect percentages of row totals. 
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Findings 

As noted previously, one of the JDRP criteria was evidence of impact. 
Evidence of impact was defined as the type of evaluation design that was 
Implemented and the type of comparison group that was used in the design. In 
Table 6, four evaluation designs— pre-experiinental, quasi-experimental, 
experimental, and qualitative—were identified by Pang, Lynch, and Lam. While 
the majority of approved submittals in 1978 and 1979 utilized either a pre- 
experimental or quasi-experimental design, the majority of approvals for the 
years 1960-83 were quasi-experimental designs. This trend changed in 1984 and 
1985 when the majority of approvals were experimental designs. As time 
passes, it appears that studies with more rigid controls are being conducted 
and submitted to JDRP. 

The type of comparison group reflects the type of evaVaation design that 
was implemented. As is expected, the majority of the approvals in the years 
1978 through 1983 used a ncnequivalent control group. The majority of 
approvals in 1984 and 1985 used a randomized control group as the comparison 
group. These data are presented in Table 5. 
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Mai* 6 
Type of Evaluation Design 



10 



Pre- Quasi- Bbcper'l 

Experimental BxDer'l Design Qualitative Total 

Approvals 24 (46) a 23 (44) 5 (10) 0 62 

Reaction* 12 (67) 6 ( 26) 1 (6) 0 18 



Re jectlons Lator 



7 (64) 6 (39) 1 (8) 0 13 



6 (46) 6 (39) 2 (16) 0 13 
Lynch 0 

Approvals 36 (18) 127 (66) 33 (17) 1 (1) 196 

Reactions 15 (13) 88 (73) 17 (14) 0 120 



Approvals 10 (43) 0 13 (57) 23 

Rejections _3 (38) _l (13) _4 (60) 8 

TOTAL 

Approvals 76 155 53 1 

Rejections 37 99 23 0 

*ttnbsrs In parent h eses reflect percentages of ion totals. 
b Lynch looked at all evaluation designs that ware presented in the 
submittals. A total of 316 evaluation designs were reported in the 232 
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Table 6 
Type of Ooeparlson Group 
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Pre- 

Self/ National experi- Nan- Randam- 

mesterv Norm mental equivalent lzed othar Total 

Fbdq 

Approvals 4 (8) 

Rejections 2 (11 

Rejections later 

Resubmitted 0 
Resubmlttals 

Ap pro v ed 1 (8) 

Lynch 0 
Approvals 
Rejections 



11 


(21) 


9 (17) 


23 


(44) 


5 


(10) 




52 


6 


(33) 


4 (22) 


5 


(28) 


1 






18 


5 


(39) 


2 (15) 


5 


(38) 


1 


(8) 




13 


3 


(23) 


2 (15) 


5 


(39) 


2 


(15) 




13 


59 


(26) 


35 (15) 


111 


(46) 


33 


(14) 




238 


29 


(24) 


15 (13) 


59 


(49) 


17 


(14) 




120 


4 


(17) 


3 (13) 






14 


(61) 


2 (9) 


23 


1 


(12) 


3 (38) 






4 


(50) 


_0 


8 


118 




73 


208 




77 




2 


485 



Approvals 
Rejections 
TOTAL 



a Ntmbers In parentheses reflect percentages of row totals. 

^Lynch looked at all title evaluation designs and types of comparison groups 
that ware reported. A total of 358 comparison groups were reported in the 232 
submittals. 
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A Mcond criterion of effectiveness was the evidence of statistical 
reliability of the effects. This was operationally defined as the 
presentation of significance testing in the submittal's narrative. Both Feng 
and Lynch looked at this criterion and found that there was no difference 
batmen approvals and disapprovals on this variable. It appears that both 
included significance testing as a part of their data presentation. 



Table 7 

Inclusion of Significance Testing 



Fang 
Approvals 
Rejections 

Rejections Later Resubmitted 
Resubmittals Approved 
Lynch 
Approvals 
Rejections 



Descriptive Significance 
Statistics Only Testing 



9 (17) 
3 (17) 
3 (23) 



4 (3) 
1 (1) 



43 (83) 
15 (83) 
10 (77) 
13 (100) 

140 (97) 
87 (99) 



Total 

52 
18 
13 
13 

144 
88 



Note: Lam did not look at this variable. 
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Interpretability of manures was another criterion of effectiveness. 
This was operationally defined as the type of instrument that was administered 
(either standardized or locally-developed) and whether or not reliability and 
validity Information was presented for the instrument. Table 8 presents the 
data on the type of Instruments that ware used by the program developers. 
Soma program administered both standardized and locally-developed 
Instruments, which explains why the number of instruments exceeds the number 
of submittals. Lynch did not pr es en t these data by approved vs. disapproved 
submittals since she found that this variable made little difference in how a 
program was reviewed by JERP. Reliability and validity information on each 
test is also omitted in this p r e s entation since Fang and Lynch defined these 
variables s o mew hat differently. 
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Fang 

Approvals 

Rejections 

Rejections Later 
Resubmitted 

Resubmlttal 
Approved 
Lynch c 
Lam 

Approvals 

Rejections 
TOTALS 



Table 6 
Types of Ins! 1 1— it a Used 

Standar- Locally- 



dized 

42 (36) b 
14 (67) 

17 (63) 

14 (58) 
250 (62) 

8 (35) 
7 (88) 
352 



Developed Modified* Other Total 



78 (64) 
7 (33) 

10 (37) 

10 (42) 
129 (32) 

15 (65) 

_1_1121 
250 



12 (3) 



12 



1 (1) 121 
21 

27 

24 

15 (4) 406 

23 

_8 

16 630 



Note: Lynch did not break these down into approvals and disapprovals. 
a Pang and Lam did not use this category. 
b Numbers in parentheses reflect percentages of row totals. 
PLynch looked at all instruments that were administered and reported. A 
total of 406 instruments were presented in the 232 submittals. 



Other effectiveness criteria ass e ss ed evidence of educational 
importance, credibility of evidence, internal validity, and external validity. 
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Because Pang, Lynch, and Lam defined and/or presented these variables somewhat 
differently, these are not discussed here. However, evidence of general iza- 
bility, as defined by the inclusion of replication data, is presented in Table 
9 below. Table 9 Indicates that replication did occur, as d emon st rated by 
data provided on more than one location or more than one time period. Lynch 
did not break these data down by approvals vs. disapprovals since she found 
that the majority of submittals included these replication data. 

Trials 9 

Inclusion of Replication Data 

Replication Occurred 
by Locations or Time Periods 



Fang 

Approvals 32 (62) a 

Rejections 11 (61) 

Rejections Later Resubmitted 7 (54) 

Resufamittal A ppro v ed 9 (69) 
Lynch 

Locations 29 

Time periods 132 
lam 

Approvals 6 (26) 

Rejections 0 

TOTAL 226 



a Numbers in parentheses reflect percentages of approved and disapproved 
submittals. 
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In addition to the effectiveness criteria, Fang, Lynch, and Lam looked at 
how clearly the submittal was written. Presentation of results appeared to be 
a critical factor in whether a submittal was approved or disappro v e d by JDRP. 
The majority of approved submittals ware either clearly written (as rated an a 
five-point scale by Fang and Lam) or were of good quality (as rated on a 
three-point scale by Lynch) . These findings are presented in Table 10. 
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Mala 10 
Clarity of Narrmtlw 



Fang 
Approvals 
Rejections 
Rejections Later 

Resubmitted 
Resutanittal 
Approved 



Approvals 
Rejections 



Not Clear at All 

1 2 



Extremely Clear 
_3 4 5 Total 



10 (19) a 19 (37) 22 (42) 1 (3) 52 
5 (28) 9 (50) 4 (22) 18 



1 (8) 6 (46) 5 (38) 1 (8) 



13 



2 (9) 



1 (8) 5 (38) 6 (46) 1 (8) 13 

3 (13) 3 (13) 10 (43) 5 (22) 23 

2 (25) 3 (38) 2 (25) 1 (13) 8 



Lynch 
Approvals 
Rejections 



Quality of Submittal 

Poor Fair Good 

5 (3) 50 (35) 89 (62) 
24 (27) 50 (57) 14 (16) 



Total 

144 
88 



'Numbers In parentheses reflect percentages of row totals. 



ERIC 



20 



18 

The importance of a well-written submittal was further illustrated by Fang and 
Lynch when they performed a stepwise discriminant analysis procedure on 
selected variables related to the JDRP review criteria. This procedure was 
initiated in order to determine whether approved submittals could be 
distinguished from disapproved submittals on the basis of JDRP review 
criteria. According to Fang's results, predictor variables included the 
clarity of the tables, a description of progr a m components, the type of 
evaluation design, the number of instruments, and the elimination of selection 
and statistical regression effects. Lynch' s results were similar; she found 
that predictor variables included the quality rating of the submittal, the 
amount of information that was included in the submittal, the type of 
evaluation design, and the cost of installation for the intervention. Both 
studies Indicate that the presentation of progr a m components and evaluation 
findings play a major role in the approval and disapproval of submittals. 
Conclusions 

1. The number of submittals has varied greatly over the eight-year period 
and is currently at an all-time low. 

2. The quality of the evaluation designs has consistently impro v ed, from 
roughly 10* in the 1970 's to over 50* in 1984-85. 

3. The content of the curricular area has varied over time and most 
likely reflects changes in priorities set by the National Diffusion Network. 

4. One of the key variables in determining whether a submittal is 
approved is the quality of the presentation. This variable was identified as 
critical in each of the three studies. 
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